{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d2451aff",
   "metadata": {},
   "source": [
    "# Evaluating LlamaIndex\n",
    "\n",
    "[LlamaIndex](https://github.com/jerryjliu/llama_index) is a data framework for LLM applications to ingest, structure, and access private or domain-specific data. Makes it super easy to connect LLMs with your own data. But in order to figure out the best configuration for llamaIndex and your data you need a object measure of the performance. This is where ragas comes in. Ragas will help you evaluate your `QueryEngine` and gives you the confidence to tweak the configuration to get hightest score.\n",
    "\n",
    "This guide assumes you have familarity with the LlamaIndex framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37a9b094",
   "metadata": {},
   "outputs": [],
   "source": [
    "# attach to the same event-loop\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abaf6538",
   "metadata": {},
   "source": [
    "## Building the `VectorStoreIndex` and `QueryEngine`\n",
    "\n",
    "To start lets build an `VectorStoreIndex` over the New York Citie's [wikipedia page](https://en.wikipedia.org/wiki/New_York_City) as an example and use ragas to evaluate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "37c4a1cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9019540b",
   "metadata": {},
   "source": [
    "load the data, build the `VectorStoreIndex` and create the `QueryEngine`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "85e75230",
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader(\"./nyc_wikipedia/\").load_data()\n",
    "vector_index = VectorStoreIndex.from_documents(\n",
    "    documents, service_context=ServiceContext.from_defaults(chunk_size=512)\n",
    ")\n",
    "\n",
    "query_engine = vector_index.as_query_engine()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13d676c0",
   "metadata": {},
   "source": [
    "Lets try an sample question to see if it is working"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a25026c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "New York City was named in honor of the Duke of York, who would become King James II of England. In 1664, King Charles II appointed the Duke as proprietor of the former territory of New Netherland, including the city of New Amsterdam, when England seized it from Dutch control. The city was then renamed New York in his honor.\n"
     ]
    }
   ],
   "source": [
    "response_vector = query_engine.query(\"How did New York City get its name?\")\n",
    "\n",
    "print(response_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b678501e",
   "metadata": {},
   "source": [
    "## Evaluating with Ragas\n",
    "\n",
    "Now that we have a `QueryEngine` for the `VectorStoreIndex` we can use the llama_index integration Ragas has to evaluate it. \n",
    "\n",
    "In order to run an evaluation with Ragas and LlamaIndex you need 3 things\n",
    "\n",
    "1. LlamaIndex `QueryEngine`: what we will be evaluating\n",
    "2. Metrics: Ragas defines a set of metrics that can measure different aspects of the `QueryEngine`. The available metrics and their meaning can be found [here](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md)\n",
    "3. Questions: A list of questions that ragas will test the `QueryEngine` against. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "145109ad",
   "metadata": {},
   "source": [
    "first lets generate the questions. Ideally you should use that you see in production so that the distribution of question with which we evaluate matches the distribution of questions seen in production. This ensures that the scores reflect the performance seen in production but to start off we'll be using a few example question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "751dc988",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_questions = [\n",
    "    \"What is the population of New York City as of 2020?\",\n",
    "    \"Which borough of New York City has the highest population?\",\n",
    "    \"What is the economic significance of New York City?\",\n",
    "    \"How did New York City get its name?\",\n",
    "    \"What is the significance of the Statue of Liberty in New York City?\",\n",
    "]\n",
    "\n",
    "eval_answers = [\n",
    "    \"8,804,000\",  # incorrect answer\n",
    "    \"Queens\",  # incorrect answer\n",
    "    \"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.\",\n",
    "    \"New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.\",\n",
    "    \"The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.\",\n",
    "]\n",
    "\n",
    "eval_answers = [[a] for a in eval_answers]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "843bddb8",
   "metadata": {},
   "source": [
    "Now lets import the metrics we will be using to evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9875132a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import (\n",
    "    faithfulness,\n",
    "    answer_relevancy,\n",
    "    context_precision,\n",
    "    context_recall,\n",
    ")\n",
    "from ragas.metrics.critique import harmfulness\n",
    "\n",
    "metrics = [\n",
    "    faithfulness,\n",
    "    answer_relevancy,\n",
    "    context_precision,\n",
    "    context_recall,\n",
    "    harmfulness,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ae4a2d1",
   "metadata": {},
   "source": [
    "Finally lets run the evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "05633cc2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [faithfulness]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████| 1/1 [00:51<00:00, 51.40s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [answer_relevancy]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████| 1/1 [00:09<00:00,  9.64s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [context_precision]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████| 1/1 [00:41<00:00, 41.21s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [context_recall]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████| 1/1 [00:34<00:00, 34.97s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [harmfulness]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████| 1/1 [00:13<00:00, 13.98s/it]\n"
     ]
    }
   ],
   "source": [
    "from ragas.llama_index import evaluate\n",
    "\n",
    "result = evaluate(query_engine, metrics, eval_questions, eval_answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f927a943",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{faithfulness': 0.7000, 'answer_relevancy': 0.9550, 'context_precision': 0.2335, 'context_recall': 0.9800, 'harmfulness': 0.0000}\n"
     ]
    }
   ],
   "source": [
    "# final scores\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "878b6b82",
   "metadata": {},
   "source": [
    "You can convert into a pandas dataframe to run more analysis on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b96311e2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>contexts</th>\n",
       "      <th>answer</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>faithfulness</th>\n",
       "      <th>answer_relevancy</th>\n",
       "      <th>context_precision</th>\n",
       "      <th>context_recall</th>\n",
       "      <th>harmfulness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the population of New York City as of ...</td>\n",
       "      <td>[Aeromedical Staging Squadron, and a military ...</td>\n",
       "      <td>\\nThe population of New York City as of 2020 i...</td>\n",
       "      <td>[8,804,000]</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.999999</td>\n",
       "      <td>0.320000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Which borough of New York City has the highest...</td>\n",
       "      <td>[co-extensive with New York County, the boroug...</td>\n",
       "      <td>\\nThe borough of Manhattan has the highest pop...</td>\n",
       "      <td>[Queens]</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.998524</td>\n",
       "      <td>0.038462</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is the economic significance of New York ...</td>\n",
       "      <td>[health care and life sciences, medical techno...</td>\n",
       "      <td>\\nNew York City is a major global economic cen...</td>\n",
       "      <td>[New York City's economic significance is vast...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.904272</td>\n",
       "      <td>0.423077</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How did New York City get its name?</td>\n",
       "      <td>[a US$1 billion research and education center ...</td>\n",
       "      <td>\\nNew York City was named in honor of the Duke...</td>\n",
       "      <td>[New York City got its name when it came under...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.929719</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is the significance of the Statue of Libe...</td>\n",
       "      <td>[(stylized I ❤ NY) is both a logo and a song t...</td>\n",
       "      <td>\\nThe Statue of Liberty is a symbol of the Uni...</td>\n",
       "      <td>[The Statue of Liberty in New York City holds ...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.942676</td>\n",
       "      <td>0.052632</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What is the population of New York City as of ...   \n",
       "1  Which borough of New York City has the highest...   \n",
       "2  What is the economic significance of New York ...   \n",
       "3                How did New York City get its name?   \n",
       "4  What is the significance of the Statue of Libe...   \n",
       "\n",
       "                                            contexts  \\\n",
       "0  [Aeromedical Staging Squadron, and a military ...   \n",
       "1  [co-extensive with New York County, the boroug...   \n",
       "2  [health care and life sciences, medical techno...   \n",
       "3  [a US$1 billion research and education center ...   \n",
       "4  [(stylized I ❤ NY) is both a logo and a song t...   \n",
       "\n",
       "                                              answer  \\\n",
       "0  \\nThe population of New York City as of 2020 i...   \n",
       "1  \\nThe borough of Manhattan has the highest pop...   \n",
       "2  \\nNew York City is a major global economic cen...   \n",
       "3  \\nNew York City was named in honor of the Duke...   \n",
       "4  \\nThe Statue of Liberty is a symbol of the Uni...   \n",
       "\n",
       "                                       ground_truth  faithfulness  \\\n",
       "0                                        [8,804,000]           1.0   \n",
       "1                                           [Queens]           0.0   \n",
       "2  [New York City's economic significance is vast...           1.0   \n",
       "3  [New York City got its name when it came under...           1.0   \n",
       "4  [The Statue of Liberty in New York City holds ...           0.5   \n",
       "\n",
       "   answer_relevancy  context_precision  context_recall  harmfulness  \n",
       "0          0.999999           0.320000             1.0            0  \n",
       "1          0.998524           0.038462             0.9            0  \n",
       "2          0.904272           0.423077             1.0            0  \n",
       "3          0.929719           0.333333             1.0            0  \n",
       "4          0.942676           0.052632             1.0            0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.to_pandas()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
